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FIELD AND BACKGROUND OF THE INVENTION 

The present invention relates to a system and method for teaching and 
training students in medical procedures, and in particular to a system and 
method for training students in the procedure of flexible gastro-endoscopy. 

Flexible gastro-endoscopy is an important medical tool for both surgical 
and diagnostic procedures in the gastro-intestinal tract. Essentially, gastro- 
endoscopy is performed by inserting an endoscope, which is a flexible tube, 
into the gastro-intestinal tract, either through the mouth or the rectum of the 
subject. The tube is manipulated by a trained physician through specialized 
controls. The end of the tube which is inserted into the subject contains a 
camera and one or more surgical tools, such as a clipper for removing tissue 
samples from the gastro-intestinal tract. The physician must maneuver the tube 
according to images of the gastro-intestinal tract received from the camera and 
displayed on a video screen. The lack of direct visual feedback from the 
gastro-intestinal tract is one factor which renders endoscopy a complex and 
difficult procedure to master. Such lack of feedback also increases the 
difficulty of hand-eye coordination and correct manipulation of the endoscopic 
device. Thus, flexible gastro-endoscopy is a difficult procedure to both 
perform and to learn. 



Currently, students are taught to perform flexible gastro-endoscopy 
according to the traditional model for medical education, in which students 
observe and assist more experienced physicians. Unfortunately, such 
observation alone cannot provide the necessary training for such complicated 

5 medical procedures. Students may also perform procedures on animals and 
human cadavers, neither of which replicates the visual and tactile sensations of 
a live human patient. Thus, traditional medical training is not adequate for 
modem technologically complex medical procedures. 

In an attempt to provide more realistic medical training for such 

10 procedures, simulation devices have been developed which attempt to replicate 
the tactile sensations and/or visual feedback for these procedures, in order to 
provide improved medical training without endangering human patients. An 
example of such a simulation device is disclosed in U.S. Patent No. 5,403,191, 
in which the disclosed device is a box containing simulated human organs. 

15 Various surgical laparoscopic procedures can be performed on the simulated 
organs. Visual feedback is provided by a system of mirrors. However, the 



system of both visual and tactile feedback is primitive in this device, and does 
not provide a true representation of the visual and tactile sensations which 
would accompany such surgical procedures in a human patient. Furthermore, 
20 the box itself is not a realistic representation of the three-dimensional structure 
of a human patient. Thus, the disclosed device is lacking in many important 
aspects and fails to meet the needs of a medical simulation device. 



Attempts to provide a more realistic experience from a medical 
simulation devices are disclosed in PCT Patent Application Nos. WO 96/16389 
and WO 95/02233. Both of these applications disclose a device for providing a 
simulation of the surgical procedure of laparoscopy. Both devices include a 
mannequin in the shape of a human torso, with various points at which 
simulated surgical instruments are placed. However, the devices are limited in 
that the positions of the simulated surgical instruments are predetermined, 
which is not a realistic scenario. Furthermore, the visual feedback is based 
upon a stream of video images taken from actual surgical procedures. 
However, such simple rendering of video images would result in inaccurate or 
unrealistic images as portions of the video data would need to be removed for 
greater processing speed. Alternatively, the video processing would consume 
such massive amounts of computational time and resources that the entire 
system would fail to respond in a realistic time period to the actions of the 
student. At the very minimum, a dedicated graphics workstation would be 
required, rather than a personal computer (PC). Thus, neither reference teaches 
or discloses adequate visual processing for real time visual feedback of the 
simulated medical procedure. 

Similarly, U.S. Patent No. 4,907,973 discloses a device for simulating 
the medical procedure of flexible gastro-endoscopy. The disclosed device also 
suffers from the deficiencies of the above-referenced prior art devices, in that 
the visual feedback system is based upon rendering of video data taken from 



actual endoscopic procedures. As noted previously, displaying such data would 
either require massive computational resources, or else would simply require 
too much time for a realistic visual feedback response. Thus, the disclosed 
device also suffers from the deficiencies of the prior art. 

A truly useful and efficient medical simulation device for minimally 
invasive therapeutic procedures such as flexible gastro-endoscopy would give 
real time, accurate and realistic visual feedback of the procedure, and would 
also give realistic tactile feedback, so that the visual and tactile systems would 
be accurately linked for the simulation as for an actual medical procedure. 
Unfortunately, such a simulation device is not currently taught or provided by 
the prior art. 

There is therefore a need for, and it would be useful to have, a method 
and a system to simulate a minimally invasive medical procedure such as 
flexible gastro-endoscopy, which would provide accurate, linked visual and 
tactile feedback to the student and which would serve as a training resource for 
all aspects of the procedure. 

SUMMARY OF THE INVENTION 

According to the present invention, there is provided a system for 
simulating a medical procedure performed on a subject, comprising: (a) a 
simulated organ; (b) a simulated instrument for performing the medical 
procedure on the simulated organ; (c) a locator for determining a location of the 



simulated instrument within the simulated organ; and (d) a visual display for 
displaying images from the medical procedure, such that the images simulate 
visual data received during the medical procedure as performed on an actual 
subject, the visual display including: (i) a three-dimensional model of the 
simulated organ, the model being divided into a plurality of segments; (ii) a 
loader for selecting at least one of the plurality of segments for display, the at 
least one of the plurality of segments being selected according to the location of 
the simulated instrument within the simulated organ; (iii) a controller for 
selecting each image from the selected segment according to the location of the 
simulated instrument; and (iv) a displayer for displaying the image according to 
the controller. 

According to preferred embodiments of the present invention, the visual 
displayer further comprises: (v) a texture mapping engine for adding texture 
mapping to the selected image substantially before the selected image is 
displayed by the displayer. Preferably, the texture mapping includes animation 
of random movement of the simulated instrument and of random movement of 
the simulated organ. 

Preferably, the texture mapping includes images which are obtained 
from a performance of the medical procedure on the actual subject by first 
recording the visual data during the performance and then selecting the images 
from the recorded visual data. 



According to other preferred embodiments of the present invention, the 
displayer further displays a graphical user interface. Preferably, the graphical 
user interface displays tutorial information for aid in performing the medical 
procedure. 

According to still other preferred embodiments of the present invention, 
the simulated organ is a gastro-intestinal tract. Preferably, the gastro- intestinal 
tract is constructed from a semi-flexible, smooth material. Also preferably, the 
simulated instrument is an endoscope, the endoscope featuring a sensor for 
determining a location of the sensor in the gastro-intestinal tract. 

According to preferred embodiments of the present invention, there is 
further provided a tactile feedback mechanism for providing simulated tactile 
feedback according to the location of the tip of the endoscope. Preferably, the 
tactile feedback mechanism is contained in the gastro-intestinal tract, and the 
gastro-intestinal tract further comprises: (a) a plurality of servo-motors; (b) a 
piston operated by each of the plurality of servo-motors, the piston contacting 
the semi-flexible material; and (c) a controller for controlling the plurality of 
servo-motors, such that a position of the piston is determined by the controller, 
and such that the position of the piston provides the tactile feedback. 

Alternatively and preferably, the tactile feedback mechanism is located 
in the endoscope, and the endoscope further comprises: (a) a guiding sleeve 
connected to the tip of the endoscope; (b) at least one ball bearing attached to 
the guiding sleeve for rolling along an inner surface of the gastro-intestinal 



tract; (c) at least one linear motor attached to the guiding sleeve; (d) a piston 
operated by the linear motor, the piston contacting the inner surface of the 
gastro-intestinal tract; and (e) a controller for controlling the linear motor, such 
that a position of the piston is determined by the controller, and such that the 
position of the piston provides the tactile feedback. 

Most preferably, the tactile feedback mechanism is operated according 
to tactile feedback obtained during the performance of the medical procedure 
on an actual subject, the tactile feedback being obtained through virtual reality 
gloves. 

According to another embodiment of the present invention, there is 
provided a method for performing a simulated endoscopic procedure, the 
method comprising the steps of: (a) providing a system for perfomiing the 
simulated endoscopic procedure, comprising: (i) a simulated gastro-intestinal 
tract; (ii) a simulated endoscope for performing the simulated endoscopic 
procedure on the simulated gastro-intestinal tract; (iii) a locator for determining 
a location of the simulated endoscope within the simulated gastro-intestinal 
tract; and (iv) a visual display for displaying images according to the simulated 
endoscope within the simulated gastro-intestinal tract, such that the images 
simulate visual data received during an actual medical procedure as performed 
on an actual subject, the visual display including: (1) a three-dimensional 
mathematical model of the simulated gastro-intestinal tract, the model being 
divided into a plurality of segments; (2) a loader for selecting at least one of the 




plurality of segments for display, the at least one of the plurality of segments 
being selected according to the location of the simulated endoscope within the 
simulated gastro- intestinal tract; (3) a controller for selecting a simulated image 
from the segment according to the location of the simulated instrument; and (4) 
a displayer for displaying the simulated image according to the controller, such 
that the simulated image is a displayed image; (b) inserting the simulated 
endoscope into the simulated gastro-intestinal tract; (c) receiving visual 
feedback according to the displayed image; and (d) receiving tactile feedback 
according to the location of the endoscope within the gastro-intestinal tract. 

According to yet another embodiment of the present invention, there is 
provided a method for displaying simulated visual data of a medical procedure, 
the method comprising the steps of: (a) recording actual video data from a 
performance of an actual medical procedure on a living human patient; (b) 
abstracting a plurality of individual images from the actual video data; (c) 
digitizing the plurality of individual images to form a plurality of digitized 
images; (d) selecting at least one of the plurality of digitized images to form a 
selected digitized image; (e) storing the selected digitized image as texture 
mapping data in a texture mapping database; (f) providing a three-dimensional 
. mathematical model of an actual human colon, the model being divided into a 
plurality of segments; (g) selecting one of the plurality of segments from the 
model for display; (h) overlaying the texture mapping data from the texture 



mapping database onto the segment of the model to form at least one resultant 
image; and (i) displaying the resultant image. 

Preferably, the texture mapping data further include animation. More 
preferably, the animation includes random movement of the simulated 
instrument and random movement of the simulated organ. 

Hereinafter, the term "simulated medical procedure" refers to the 
simulation of the medical procedure as performed through the system and 
method of the present invention. Hereinafter, the term "actual medical 
procedure" refers to the performance of the medical procedure on an actual, 
living human patient with an actual endoscope, such that the medical procedure 
is "real" rather than "simulated". 

Hereinafter, the term "endoscopy" includes, but is not limited to, the 
procedure of flexible gastro-endoscopy, as previously described, and medical 
diagnostic and surgical procedures in which an endoscope is inserted into the 
mouth or the rectum of the subject for manipulation within the gastro-intestinal 

tract of the subject. Hereinafter, the term "subject" refer s to the human or 

lower mammal upon which the method and system of the present invention are 
performed or operated. Hereinafter, the term "student" refers to any human 
using the system of the present invention, being trained according to the present 
invention or being taught according to the present invention including, but not 
limited to, students attending medical school or a university, a medical doctor, a 
trained gastro-enterologist or other trained medical specialist. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better 
understood from the following detailed description of a preferred embodiment 
5 of the invention with reference to the drawings, wherein: 

FIG. 1 is an exemplary illustration of the system for medical simulation 
according to the present invention; 

FIG. 2 is an exemplary illustration of a screen display according to the 
present invention; 

10 FIG. 3 is a schematic block diagram of an exemplary visual processing 

and display system according to the present invention; 

FIG. 4 is a schematic block diagram of an exemplary tutorial system 
according to the present invention; 

FIGS. 5 A and 5B illustrate an exemplary simulated gastro-intestinal tract 
1 5 according to the present invention; 

FIGS. 6A-C illustrate various aspects of one embodiment of the force- 



feedback system according to the present invention; and 

FIGS. 7A-7D illustrate a second embodiment of the force-feedback 
system according to the present invention. 
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BRIEF DESCRIPTION OF THE INVENTION 
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The present invention includes a method and a system to simulate the 
medical procedure of endoscopy, particularly of flexible gastro-endoscopy. 
The system is designed to simulate the actual medical procedure of endoscopy 
as closely as possible by providing both a simulated medical instrument, and 
tactile and visual feedback as the simulated procedure is performed on the 
simulated patient. Although the discussion is directed toward the medical 
procedure of endoscopy, the present invention could also be employed to 
simulate other types of minimally invasive medical procedures. 

The system of the present invention features both a physical model and a 
virtual model for the simulation of the medical procedure of endoscopy. The 
physical model includes a mannequin into which the simulated endoscope is 
inserted. A simulated gastro-intestinal tract is located within the mannequin, 
which includes a simulated rectum and a simulated colon. The simulated 
endoscope is inserted into the simulated rectum and then into the simulated 
colon. The simulated gastro-intestinal tract provides realistic tactile feedback 
according to the movement of the simulated endoscope within the simulated 
colon. 

The virtual model provides a "virtual reality" for the simulation of 
images from the endoscope. In an actual endoscopic medical procedure, a 
camera at the tip of the actual endoscope returns images from the colon of the 
human patient. These images are then viewed by the physician performing the 
endoscopic procedure, thereby providing visual feedback to the physician. The 
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system of the present invention provides a "virtual reality* 5 for the realistic 
simulation of this visual feedback. This virtual reality enables the real-time 
display of realistic images of the gastro-intestinal tract on a video monitor 
according to the manipulations of the simulated endoscope, preferably in such a 
5 manner that the tactile and visual feedback are linked as they would be in a 
human patient. 

The virtual reality has two main components: a three-dimensional, 
mathematical model of the colon, and a database of enhanced digitized images 
derived from actual visual data obtained from an actual endoscopic procedure. 
10 These two components are combined to provide realistic visual feedback bo- 
using the enhanced images as texture mapping to overlay the mathematical 
model of the colon, thereby closely simulating images obtained from the actual 
procedure. 

The virtual reality of the colon is particularly advantageous for 
15 simulating images because it does not rely on video streams, which require 
massive computational power for real-time display of visual feedback. In 
addition, video streams provide only a predetermined flow of images and 
cannot provide visual data with six degrees of freedom in real time. 
Furthermore, the virtual reality of the present invention does not rely merely on 
20 a mathematical model of the colon, which cannot capture the irregularities and 
subtle visual features of an actual colon from a human patient. Thus, the virtual 




reality of the colon provides the best simulation of realistic images in real time 
for visual feedback. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

An object of this invention is the provision of a method and a system to 
simulate the procedure of endoscopy, particularly of flexible gastro-endoscopy. 
The system includes a mannequin into which the simulated endoscope is 
inserted. Visual feedback is provided through a video monitor, which displays 
realistic images in real time, according to the manipulations of the simulated 
endoscope. Realistic tactile feedback is also provided, preferably in such a 
manner that the tactile and visual feedback are linked as they would be in a 
human patient. Another object of the present invention is the provision of a 
tutorial system for training students and testing their performance. Thus, the 
system and method of the present invention provide a realistic simulation of the 
medical procedure of endoscopy for training and testing students. 

The principles and operation of a method and a system for 
communication according to the present invention may be better understood 
with reference to the drawings and the accompanying description, it being 
understood that these drawings are given for illustrative purposes only and are 
not meant to be limiting. 

Referring now to the drawings, Figure 1 depicts an exemplary, 
illustrative system for medical simulation according to the present invention. A 
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system 10 includes a mannequin 12 representing the subject on which the 
procedure is to be performed, a simulated endoscope 14 and a computer 16 with 
a video monitor 18. A student 20 is shown interacting with system 10 by 
manipulating simulated endoscope 14 within mannequin 12. As further 
illustrated in Figures 5A and 5B below, mannequin 12 includes a simulated 
organ into which simulated endoscope 14 is inserted. As student 20 
manipulates simulated endoscope 14, tactile and visual feedback are determined 
according to the position of endoscope 14 within the simulated organ (not 
shown). The visual feedback are provided in the form of a display on video 
monitor 18. The necessary data calculations are performed by computer 16, so 
that realistic tactile and visual feedback are provided to student 20. 

Figure 2 is an exemplary illustration of a screen display shown on 
monitor 18. A screen display 22 includes a feedback image 24. Feedback 
image 24 represents the visual image as seen if the endoscope were inserted 
into a living human patient. Feedback image 24 is an accurate and realistic 
simulation of the visual data that would be received from that portion of the 
gastro-intestinal tract in the living human patient. Although feedback image 24 
is shown as a static image, it is understood that this is for illustrative purposes 
only and the actual visual feedback data would be in the form of a substantially 
continuous flow of simulated images based upon actual video stream data 
obtained from an actual endoscopic procedure. Thus, the flow of images 
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represented by feedback image 24 gives the student (not shown) realistic visual 
feedback. 

In addition, screen display 22 preferably includes a number of GUI 
(graphic user interface) features related to the preferred tutorial functions of the 
5 present invention. For example, a tracking display 26 explicitly shows the 

location of the simulated endoscope within the simulated gastro-intestinal tract. 
Tracking display 26 includes a schematic gastro-intestinal tract 28, into which a 
schematic endoscope 30 has been inserted. Preferably, tracking display 26 can 
be enabled or disabled, so that the student can only see tracking display 26 if 
10 the tracking function is enabled. 

Additional, optional but preferred features of screen display 22 include 
the provision of a "help" button 32, which upon activation could cause the 
display of such helpful information as a guide to the controls of the endoscope. 
Similarly, a preferred "hint" button 34 would give the student one or more 
1 5 suggestions on how to continue the performance of the medical procedure. A 
preferred "patient histor y " button 36 w ould ca use screen display 22 to show 



information related to one of a selection of simulated "patient histories", which 
could be of help to the student in deciding upon a further action. Finally, a 
preferred "performance" button 38 would cause screen display 22 to display a 
20 review and rating of the performance of the student. All of these functions are 
part of the preferred embodiment of a tutorial system for training a student in 
the medical procedure of endoscopy, as described in further detail in Figure 4. 
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Figures 3A and 3B are schematic block diagrams of an exemplary visual 
processing and display system and method according to the present invention. 
Figure 3 A is a flow chart of the method for visual processing and display 
according to the present invention, and is intended as a summary of the method 
5 employed by the system of Figure 3B. Further details concerning particular 
aspects of the method are described below with reference to Figure 3B. In step 
one, actual video data are recorded onto videotape during the performance of 
the actual medical procedure of endoscopy on a living human patient. In step 
two, individual images are abstracted with a framegrabber and then digitized. 
10 In step three, the digitized images are preferably selected for clarity and lack of 
visual artifacts, and are then stored in a texture mapping database. More 
preferably, the digitized images are enhanced before being stored. Most 
preferably, the texture mapping also include animation. Such animation could 
simulate effects such as random vibration of the tissue of the colon or of the 
1 5 endoscope, as well as such events as liquid flowing downward due to the 
influence of gravity. 

In step four, a portion or "chunk" of a three-dimensional mathematical 
model of the human colon is loaded for display. In step five, texture mapping 
from the database is overlaid onto the chunk of the model. Preferably, such 
20 texture mapping includes both the digitized images and additional animation. 
In step six, the resultant images are displayed. As noted previously, the images 
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are displayed in a continuous flow according to the location of the simulated 
endoscope within the simulated gastrointestinal tract. 

Figure 3B shows the visual processing and display system according to 
the present invention in more detail. A visual processing and display system 40 
includes screen display 22 for displaying the processed visual data. The visual 
data are constructed as follows. First, data are recorded from actual gastro- 
endoscopic procedures onto videotape, as shown in a recording block 42. The 
data are preferably stored on Super- VHF videotape in order to obtain the 
highest quality representation of the visual images displayed on the screen 
during the actual endoscopic procedure, as shown in block 44. Next, at least a 
portion of the frames of the videotape, and preferably substantially all the 
frames, are abstracted individually by a frame-grabber 46 to form digitized 
images. Individual digitized images can then be selected for clarity and lack of 
artifacts such as reflections from the endoscopic apparatus itself The images in 
the selected frames are then preferably enhanced and added to a texture 

mapping da taba se 48. 

Preferably, two types of texture mapping are stored in the database. The 
first type of texture mapping is intended to enhance the realistic visual aspects 
of the images, for example by removing visual artifacts. The second type of 
texture mapping is intended to simulate the behavior of a live organ and a real 
endoscope, as represented by block 50. During actual endoscopic procedures 
on a living human patient, the tissue of the colon moves somewhat, and the 
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endoscope itself vibrates and wobbles. This movement is simulated visually by 
the addition of random animation of the images, and also by the addition of 
such effects as liquid flowing downward due to the influence of gravity. Such 
animation enhances the realistic nature of the visual representation of the colon. 

5 In order for the enhanced images to be correctly displayed, the images 

must correspond to the manipulation and location of the simulated endoscope 
within the simulated colon. In particular, the texture mapping of the images 
should correspond to the location of the endoscope within the colon. Such 
correspondence between the location of the endoscope within the colon and the 

10 texture mapping is provided by a texture mapping engine 52. The texture 

mapping data is then readily accessed by the display portion of visual system 
40, as shown by block 54. 

However, as noted for previous prior art devices, simply reproducing the 
selected enhanced frames in a massive video stream would quickly overwhelm 

15 the computational resources and cause the visual display to become 

unsynchronized from the physical location of the simulated endoscope. 



Furthermore, such a video stream would not enable the correct display of 
images according to the movement of the endoscope, which preferably has six 
degrees of freedom. Thus, mere reproduction is not sufficient to ensure 
20 realistic images, even when mapped onto a three-dimensional surface. 

Preferably, visual processing and display system 40 includes a three- 
dimensional mathematical model of at least a portion of the gastro-intestinal 
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tract 56. For the purposes of discussion, model 56 is herein described as a 
three-dimensional model of the colon, it being understood that this is not meant 
to be limiting in any way. An important feature of model 56 is the division of 
the modeled colon into a plurality of segments, preferably six segments 58. 
The division is made linearly, since the spatial movement of the simulated 
endoscope is limited. In other words, the simulated endoscope cannot "jump" 
from one portion of the colon to another, but must instead proceed in a linear 
fashion along the simulated colon. In addition, the simulated endoscope can 
only be moved at a finite speed through the simulated colon. Thus, the 
endoscope must pass through each segment 58 of the three-dimensional model 
of the colon in sequence at a known, limited speed. 

The consequences of such a division of model 56 into six segments 58 is 
that only one segment 58 needs to be processed in any given moment, although 
a plurality of such segments 58 could be processed substantially 
simultaneously if the computing resources were available. Furthermore, the 
division reduces the visual processing into a much more manageable task, since 
model 56 includes a total of about 282,000 polygons in the preferred 
embodiment, but each segment 58 has only about 44,000 polygons. 

As the simulated endoscope moves along the simulated colon, the 
location of the endoscope is given to a locator 60, described in further detail 
below. Locator 60 then instructs an object loader 62 to load the relevant 
segment 58 for access by visual system 40, as shown in block 54 and 
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previously described. In the preferred embodiment shown, preferably three 
segments 58 are ready for access by object loader 62 at any given moment. The 
specific segment 58 in which the endoscope is currently located is preferably 
held in DRAM or RAM. in combination with the texture mapping described 
5 previously. The next segment 58 and the preceding segment 58 are also kept 
easily accessible, although not necessarily in RAM or DRAM. 

Preferably, the display of each image from specific segment 58 into 
which the simulated endoscope has entered is optimized by a segment 
optimizer 64. Segment optimizer 64 receives information from locator 60, as 
10 well as the series of images obtained from overlaying the texture mapping onto 
the relevant segment 58, and then feeds each specific image to a display 
manager 66 for display on screen display 22. 

In addition, display manager 66 is assisted by a real-time viewer 68, 
preferably implemented in Direct 3D™ (Microsoft Inc., Seattle, Washington). 
1 5 Real-time viewer 68 provides the necessary software support to communicate 
with a graphics card 70 for actual display of the images on screen display 22. 



Although graphics card 70 can be of any suitable manufacture, preferably 
graphics card 70 has at least 8, and more preferably 16, Mb of VRAM for 
optimal performance. An example of a suitable graphics card 70 is the 3Dfx 
20 Voodoo Rush™ card. Preferably, the performance of real-time viewer 68 is 
enhanced by a math optimizer 72, preferably implemented in Visual C++. 
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The interaction between segment optimizer 64 and display manager 66 
on the one hand, and locator 60 on the other, is provided through a software 
interface 74, preferably implemented as a Direct Plug-in™ (Microsoft Inc., 
Seattle, Washington). Software interface 74 enables locator 60 to communicate 
with the other components of visual system 40, in order to provide information 
regarding the location of the endoscope within the colon. 

In preferred embodiments of the present invention, locator 60 includes a 
sensor 76, which can be obtained from Ascension Technology Corp., for 
example. Sensor 76 senses positional information from within a simulated 
organ 77, which is described herein as a colon for the purposes of discussion 
and is not meant to be limiting. Sensor 76 is controlled by a control unit 82. 
The positional information is then relayed to a CPU controller 78, which is 
connected to a servo-motor 80 (Haydon Switch and Instrument Co.). As the 
simulated endoscope moves through the colon, the endoscope contacts different 
portions of the colon (not shown; see Figures 5 and 6 below). Tactile feedback 
is provided by each servo-motor 80 in turn, which manipulates the mate rial of 
the colon. 

Visual system 40 also includes a user interface 84, preferably 
implemented in Visual C++. User interface 84 includes the GUI features 
described previously for Figure 2. In addition, user interface 84 enables visual 
system 40 to interact with the preferred feature of a network interface 86, for 
example, so that other students can view screen display 22 over a network. 



User interface 84 also permits the tutorial functions of at least one, and 
preferably a plurality of, tutorial modules 88 to be activated. Tutorial module 
88 could include a particular scenario, such as a subject with colon cancer, so 
that different types of diagnostic and medical challenges could be presented to 
the student. The student would then need to respond correctly to the presented 
scenario. 

An example of the tutorial system is illustrated in more detail in the 
block diagram of Figure 4. A tutorial system 90 starts as shown in block 92. 
Next, the user must select whether actual interaction with the simulated 
endoscope is desired, or if the user prefers to receive tutoring in the theory of 
endoscopy, as shown in a block 94. The next display asks if the user is new, as 
shown in a block 96. If the answer is "yes", the user is requested to enter 
certain information, as shown by block 98. If the answer is "no", the user is 
requested to enter identification information, such as user name or 
identification number, as shown in block 100. 

Next, the user must select the type of tutoring. For example, the user 
could select tutoring by subject 102, tutoring by procedures 104 or tutoring by 
case studies 106. Tutoring by subject 102 includes, but is not limited to, such 
subjects as basic manipulation of the endoscope, biopsy and polypectomy. 
Tutoring by subject 102 includes on-screen support, as shown in block 108. 

Tutoring by case studies 106 can be selected both according to case 
number and according to the level of the desired cases, such as beginner, 
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intermediate and expert. Preferably, individual case studies could be created by 
a teacher or professor, by combining features of various stored cases. For 
example, a professor could create a case history appropriate for a 20 year old 
male with colitis, so that the student would then be able to practice endoscopy 
5 on such a patient. Thus, tutoring system 90 preferably has the flexibility to 
enable many different types of "patients" to be studied. 

If desired, on-screen support can be provided for both tutoring by case 
studies 106 and tutoring by procedures 104. as shown in block 110. If on- 
screen support is not desired, the user can indicate whether the tutoring session 
10 is actually an official test, as shown in block 112. Thus, tutoring system 90 
includes both the ability to teach and the ability to test the student. 

Figures 5A and 5B illustrate the mechanical aspects of an exemplary 
simulated gastro-intestinal tract according to the present invention. A cut- 
away view of a mannequin 114 is shown in Figure 5 A. Preferably, mannequin 
15 114 is about one meter wide, which is within the dimensions of an actual 
_ human subject. A simulated gastro-intestinal tract 116 is shown within 
mannequin 114. For the purposes of clarity, simulated gastro-intestinal tract 
116 includes only the colon, it being understood that this is not meant to be 
limiting in any way. Simulated gastro-intestinal tract 116 is connected to a 
20 transmitter 118 and a signal processing device 120, also placed within 

mannequin 114. As shown, a simulated endoscope 122 can be inserted into 
mannequin 114 through an opening 124. In this case, since the simulation is for 
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endoscopy of the colon of the subject, opening 124 simulates the rectum of the 
subject. 

Simulated endoscope 122 can be maneuvered left, right, up and down. 
Preferably, simulated endoscope 122 is about 1 800 cm long, similar to the 
5 length of a real endoscope. Also preferably, the diameter of the tip of simulated 
endoscope 122 is about 13.4 mm, while the remainder of endoscope 122 has a 
diameter of about 10.2 mm, again similar to the dimensions of a real 
endoscope. 

Once simulated endoscope 122 is inserted into simulated gastro- 
1 0 intestinal tract 116, sensor 76 on the tip of simulated endoscope 122 is able to 
detect the location of simulated endoscope 122. Sensor 76 preferably has three 
degrees of freedom, more preferably six degrees of freedom for effective 
simulation of manipulation of endoscope 122. If sensor 76 has six degrees of 
freedom, the detected directions of orientation include the Cartesian coordinates 
15 X, Y, Z, as well as roll, elevation and azimuth. In addition, sensor 76 preferably 
includes a sensor transmitter 126, so that the precise angle and location of 
sensor 76 can be determined relative to gastro-intestinal tract 116. Sensor 
transmitter 126 transmits data to signal processing device 120, which then 
analyzes and processes the signal. The processed signal is then given to 
20 transmitter 118 for transmission to an electronics unit 128 and a DC drive unit 
130. The signal is converted by DC drive unit 130 and passed to electronics 
unit 128. Electronics unit 128 then sends the position and orientation of sensor 
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76 to software interface 74, so that the remainder of the display system is able 
to use the information to display the correct images on display screen 22 for 
visual feedback. 

The present invention provides both visual feedback and tactile 
5 feedback. Tactile feedback can be provided through the exertion of forces on 
simulated endoscope 122 by simulated gastro-intestinal tract 116, as shown in 
Figures 6A-6C. Alternatively, tactile feedback could be provided by the 
mechanical action of simulated endoscope 122 5 as shown in Figures 7A-7D. 
For the first embodiment, preferably simulated gastro-intestinal tract 116 is 
10 constructed from semi-flexible material, which gives the feel of a smooth and 
wet material. Of course, the actual sensations of sliding along a semi-flexible, 
smooth, wet material can also be provided through the mechanism of 
endoscope 122 itself, as in the second embodiment. 

An additional embodiment of gastro-intestinal tract 116, in which tract 
15 116 is placed within a box 132 rather than within mannequin 114, is shown in 
Figure 5B. The advantage of box 132 is that box 132 could serve to contain 
any radiowaves, so that the mechanism of gastro-intestinal tract 116 could be 
controlled by transmission of radiowaves, for example. Since certain medical 
equipment is highly sensitive to these radiowaves, they would need to remain 
20 within mannequin 114. Box 132 would therefore act to insulate gastro- 
intestinal tract 116 from the external environment outside the mannequin. 
Details of gastro-intestinal tract 116 are more readily seen in Figure 6 A, it 
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being understood that Figures 5 A, 5B and 6A illustrate the same gastro- 
intestinal tract 116. 

Figure 6 A shows gastro-intestinal tract 116 according to the first * 
embodiment, in which tactile feedback is provided by forces acting on 
5 simulated endoscope 122 by a mechanism contained within gastro-intestinal 
tract 116 itself. Simulated gastro-intestinal tract 116 is made from a semi- 
flexible material. A plurality of motion boxes 134 are disposed at intervals 
along the outer surface of gastro-intestinal tract 116. For the purposes of 
illustration, seven motion boxes 134 are shown. Each motion box 134, shown 
10 in greater detail in Figure 6B, has at least one, and preferably a plurality of, 
servo-motors 80, preferably linear motors. 

Each servo-motor 80 is connected to a piston 136. The detail of piston 
136 is shown enlarged in Figure 6B. Each piston 136 is connected to a foot 
138, which contacts a portion of the material of the external surface of gastro- 
15 intestinal tract 116. Preferably, foot 138 is actually attached to the portion of 
the material of the external surface, for easier manipulation of the material. 

Preferably, there are two different types of pistons 136. The first type, 
of which two are shown for illustrative purposes, is a vertical force piston 140 
for causing vertical movement of a portion of the external surface of gastro- 
20 intestinal tract 116. The second type, of which one is shown for illustrative 

purposes, is a horizontal force piston 142 for causing horizontal movement of a 
portion of the external surface of gastro-intestinal tract 116. In the preferred 
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embodiment shown, servo-motor 80 is an oscillating motor placed directly 
against the material of gastro-intestinal tract 116, so that horizontal force piston 
142 includes the motor alone, without a structure similar to vertical force piston 
140. Since each piston 136 has an associated servo-motor 80, the necessary 
vertical and horizontal movement of the external surface of gastro-intestinal 
tract 116 can be precisely determined by the activity of servo-motor 80. 

Each piston 136, or preferably attached foot 138, contacts the material of 
gastro-intestinal tract 116 in order to manipulate this material to exert a force 
against the endoscope (not shown). For example, as shown in Figure 6B, a first 
vertical force piston 144 could be moved closer to servo-motor 80, while a 
second vertical force piston 146 is moved away from servo-motor 80. These 
movements alter the position of the material of gastro-intestinal tract 116, 
causing forces to be exerted against the simulated endoscope similar or 
identical to those felt during an actual endoscopic procedure. In addition, 
horizontal force piston 142, which is preferably an oscillating servo-motor 
alone as shown, moves horizontally to provide more delicate fine-tuning of the 
tactile feedback sensations. Since servo-motors 80 are disposed over the three- 
dimensional surface of gastro-intestinal tract 116, the force on the endoscope 
can be exerted in three dimensions. 

The activity of servo-motor 80 is in turn controlled by digital controller 
82, Digital controller 82 can be a card inserted within the PC computer which 
is performing the requisite calculations required for the simulation of the 
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medical process. Software operated by the PC computer uses positional and 
orientation information from sensor 76 on simulated endoscope 122 to 
determine the position of simulated endoscope 122. Next, the software sends 
instructions to digital controller 82 according to the desired tactile sensations 
5 which should be felt by the operator of simulated endoscope 122 at that 
particular position within simulated gastro-intestinal tract 116. Digital 
controller 82 then causes at least one servo-motor 80 to move the associated 
piston 136 as necessary to provide the tactile feedback sensations. 

Digital controller 82 can be connected to servo-motors 80 through some 
10 type of radiation, such as infra-red light. However, the limitations on radiation 
of certain wavelengths, such as radiowaves, within the hospital or medical 
environment, make a connection by an actual wire running from digital 
controller 82 to each servo-motor 80 more preferable. In the exemplary 
embodiment shown in Figure 6B, each servo-motor 80 is connected to a motion 
1 5 box controller 144 by a wire. Motion box controller 144 is then preferably 
connected to digital controller 82 by a single wire (not shown). This 
configuration limits the number of individual connections made to digital 
controller 82 for greater efficiency. 

Figure 6C shows an enlarged cut-away view of servo-motor 80, which as 
20 noted previously is preferably a linear motor. Preferably, servo-motor 80 is 
about 100 mm wide and 45 mm tall. 
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Figures 7A-7D show a second embodiment of the mechanism for 
providing tactile feedback. In this embodiment, the mechanism is contained 
within the simulated endoscope itself, rather than the simulated gastro-intestinal 
tract. Similar to the previous embodiment, the simulated gastro-intestinal tract 

5 could be contained within a substantially life-size mannequin with an opening 
for simulating the rectum. Furthermore, from the viewpoint of the student or 
other individual operating the simulated endoscope, both embodiments should 
give a suitable simulation of the medical procedure. However, as detailed 
below, the actual mechanism of providing the tactile portion of the simulation 

10 differs. 

Figure 7A shows the second embodiment of a simulated endoscope 146. 
The movements and actions of simulated endoscope 146 are controlled through 
a set of controls 148. The tip of simulated endoscope 146 is contained within a 
guiding sleeve 150. Guiding sleeve 150, shown in greater detail in Figure 7B, 

1 5 preferably remains within the simulated gastro-intestinal tract (not shown; see 
Figure 7C) in order to maintain a realistic visual appearance of simulated 
endoscope 146 before insertion into the mannequin (not shown). Preferably, 
the tip of endoscope 146 has a metal bracket 152 attached, which could be 
labeled with the word "sample" or with another label in order to clarify that 

20 endoscope 146 is only a simulation and not an actual medical instrument. The 
inside of guiding sleeve 150 is preferably magnetized, for example with an 
electric current. Thus, when the tip of endoscope 146 is inserted in the 
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mannequin, metal bracket 152 is attracted to guiding sleeve 150 so that guiding 
sleeve 150 remains attached to the tip of endoscope 146. 

Guiding sleeve 150 has at least one, and preferably a plurality of, ball 
bearings 154 attached to the exterior surface of guiding sleeve 150. In addition, 
guiding sleeve 150 has at least one, and preferably a plurality of, attached 
plungers 156. As shown in the detailed view in Figure 7B, one end of guiding 
sleeve 150 preferably features a section of flexible material 158. As shown, the 
tip of endoscope 146 is preferably inserted through guiding sleeve 150. The tip 
of endoscope 146 features sensor 76, as for the previous embodiment of the 
simulated endoscope. 

Figure 7C shows simulated endoscope 146 after insertion within the 
second embodiment of a simulated gastro-intestinal tract 160. Simulated 
gastro-intestinal tract 160 is preferably constructed from a rigid material. In 
addition, simulated gastro-intestinal tract 160 preferably has the general 
anatomical shape and features of an actual gastro-intestinal tract for two 
reasons. First, the general anatomical shape can be more easily contained 
within the mannequin because of its bends and turns. Second, the general 
anatomical shape can provide gross tactile feedback. For example, as any 
endoscope is inserted more deeply into the colon, the shape of the colon cause.-, 
the tactile sensations to be altered as the endoscope moves around a bend in the 
colon. Thus, the general anatomical shape is more useful for an effective 
simulation. 
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As endoscope 146 moves within simulated gastro-intestinal tract 160, 
guiding sleeve 150 enables the operator to receive tactile feedback as follows. 
Ball bearings 154 roll along the interior surface of gastro-intestinal tract 160. 
Each ball bearing 154 has five degrees of freedom for movement. Each plunger 
156 is connected to a linear motor 162, as shown in cross-section in Figure 7D. 
Linear motor 162 is controlled in a similar fashion as the servo-motor of the 
previous embodiment. Upon receipt of signals from the computer, linear motor 
162 causes plunger 156 to move vertically, thereby causing the operator of 
simulated endoscope 146 to receive tactile feedback sensations. Thus, guiding 
sleeve 150 causes tactile feedback to be transmitted back through endoscope 
146. 

In addition, as noted above guiding sleeve 150 preferably has section of 
flexible material 158. Section of flexible material 158 causes the tip of 
endoscope 146 to encounter some resistance under certain circumstances, such 
as when the tip is bent back on itself. Thus, section of flexible material 158 

restrains^movementjaflhe Jip^fr.om certain-angles „ — — 

The particular advantages of this second embodiment is that the majority 
of tactile sensations are determined by the endoscope itself, so that they can be 
more easily controlled from the PC computer. Furthermore, such anatomical 
features as a fistula can be added according to instructions from the computer, 
without the necessity of changing the physical model of the simulated gastro- 
intestinal tract. Additionally, under certain circumstances the tissue of the 
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actual colon will force the endoscope backwards, a situation which can be more 
easily replicated in the second embodiment. Thus, the second embodiment of 
the simulated gastro-intestinal tract and endoscope is more flexible in terms of 
replicating a greater variety of anatomical features and conditions. 
5 In order to accurately replicate the tactile sensations of an actual 

endoscope during a medical procedure, these sensations must be accurately 
obtained during an endoscopic procedure in an actual living patient. For 
example, such tactile sensations could be collected from a physician performing 
the endoscopic procedure while wearing virtual reality gloves, such as the 
10 DataGloves™ Tracking VR System (Greenleaf Medical Systems). These 

gloves are known for being able to register data regarding tactile sensations and 
feedback as experienced by the physician during the actual endoscopic 
procedure. Such actual data are important because the tactile sensations change 
during the course of the procedure. For example, correlation between the 
1 5 movement of the endoscope and the visual display is gradually decreased as the 
endoscope is inserted deeper into the gastro-intestinal tract. Thus, the 



collection of actual data is an important step in the provision of an accurate, 
realistic endoscopic simulator. 

Finally, according to another preferred embodiment of the present 
20 invention there is provided a simulated biopsy device (not shown). This biopsy 
device would simulate the actual biopsy device used to retrieve tissue samples 
from the gastro-intestinal tract during endoscopy. The actual biopsy device is 
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contained within the endoscope. When the operator of the endoscope wishes to 
take a sample, the biopsy device emerges from the tip of the endoscope, at 
which point it is visible on the display screen. The jaws of the biopsy device 
are then opened and pushed onto the tissue. The jaws are then closed, and the 
biopsy device retracted. The removal of the tissue causes pools of blood to 
appear as the remaining tissue bleeds. 

Similarly, the simulated biopsy device will only appear on the display 
screen of the present invention when the operator of the simulated endoscope 
causes the simulated biopsy device to emerge. The jaws of the biopsy device 
are preferably rendered as animation, more preferably in relatively high 
resolution because the jaws are small, so that a high resolution would not prove 
unduly taxing for the PC computer. The bleeding of the tissue and the resultant 
pools of blood will also be animated. 

It will be appreciated that the above descriptions are intended only to 
serve as examples, and that many other embodiments are possible within the 
spirit and the scope of the present invention. 



WHAT IS CLAIMED: 



1 . A system for performing a simulated medical procedure, 
comprising: 

(a) a simulated organ; 

(b) a simulated instrument for performing the simulated medical 
procedure on said simulated organ; 

(c) a locator for determining a location of said simulated instrument 
within said simulated organ; and 

(d) a visual display for displaying images according to said location 
of said simulated instrument within said simulated organ, such 
that said images simulate actual visual data received during an 
actual medical procedure as performed on an actual subject, said 
visual display including: 

(i) a three-dimensional mathematical model of said simulated 
organ, said model being divided into a plurality of 
segments; 

(ii) a loader for selecting at least one of said plurality of 

segments for display, said at least one of said plurality of 
segments being selected according to said location of said 
simulated instrument within said simulated organ; 



(iii) a controller for selecting a simulated image from said 
segment according to said location of said simulated 
instrument; and 

(iv) a displayer for displaying said simulated image. 



2. The system of claim 1, wherein said visual displayer further 
comprises: 

(v) a texture mapping database for storing texture mapping data; and 

(vi) a texture mapping engine for overlaying said simulated image 
with said texture mapping data substantially before said simulated 
image is displayed by said displayer. 



3. The system of claim 2, wherein said texture mapping is animation 
of random movement of said simulated instrument and random movement of 
said simulated organ. 



4. The system of claim 1, wherein said texture mapping includes 
images obtained from performing said actual medical procedure on said actual 
subject. 



36 

5. The system of claim 4, wherein said images are obtained by first 
recording said visual data during said performance and then selecting said 
images from said recorded visual data. 

6. The system of claim 1 3 wherein said displayer further displays a 
graphical user interface. 

7. The system of claim 6, wherein said graphical user interface 
displays tutorial information for aid in performing the medical procedure. 

8. The system of claim 1, wherein said simulated organ is a gastro- 
intestinal tract. 

9. The system of claim 8, wherein said gastro-intestinal tract is 
constructed from a semi-flexible, smooth material. 

10. The system of claim 8, wherein said simulated instrument is an 
endoscope, said endoscope featuring a sensor for determining a location of said 
sensor in said gastro-intestinal tract. 
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1 1 . The system of claim 10, further comprising a tactile feedback 
mechanism for providing simulated tactile feedback according to said location 
of said tip of said endoscope. 

12. The system of claim 1 1, wherein said tactile feedback mechanism 
is contained in said gastro-intestinal tract, and said gastro-intestinal tract further 
comprises: 

(a) a plurality of servo-motors; 

(b) a piston operated by each of said plurality of servo-motors, said 
piston contacting said semi-flexible material; and 

(c) a controller for controlling said plurality of servo-motors, such 
that a position of said piston is determined by said controller, and 
such that said position of said piston provides said tactile 
feedback. 



13. The system of claim 11, wherein said tactile feedback mechanism 
is located in said endoscope, and said endoscope further comprises: 

(a) a guiding sleeve connected to said tip of said endoscope; 

(b) at least one ball bearing attached to said guiding sleeve for rolling 
along an inner surface of said gastro-intestinal tract; 

(c) at least one linear motor attached to said guiding sleeve; 




(d) a piston operated by said linear motor, said piston contacting said 
inner surface of said gastro-intestinal tract; and 

(e) a controller for controlling said linear motor, such that a position 
of said piston is determined by said controller, and such that said 
position of said piston provides said tactile feedback. 

14. The system of claim 1 1, wherein said tactile feedback mechanism 
is operated according to tactile feedback obtained during said performance of 
the medical procedure on an actual subject, said tactile feedback being obtained 
through virtual reality gloves. 

15. A method for performing a simulated endoscopic procedure, 
comprising the steps of: 

(a) providing a system for performing the simulated endoscopic 
procedure, comprising: 

(i) a simulated gastro-intestinal tract; 

(ii) a simulated endoscope for performing the simulated 
endoscopic procedure on said simulated gastro-intestinal 
tract; 

(iii) a locator for determining a location of said simulated 
endoscope within said simulated gastro-intestinal tract; and 
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(iv) a visual display for displaying images according to said 
simulated endoscope within said simulated gastro- 
intestinal tract, such that said images simulate visual data 
received during an actual medical procedure as performed 
on an actual subject, said visual display including: 

(1) a three-dimensional mathematical model of said 
simulated gastro-intestinal tract, said model being 
divided into a plurality of segments; 

(2) a loader for selecting at least one of said plurality of 
segments for display, said at least one of said 
plurality of segments being selected according to 
said location of said simulated endoscope within 
said simulated gastro-intestinal tract; 

(3) a controller for selecting a simulated image from 
said segment according to said location of said 
simulated instrument; and ... . 

(4) a displayer for displaying said simulated image 
according to said controller, such that said simulated 
image is a displayed image; 

inserting said simulated endoscope into said simulated gastro- 
intestinal tract; 

receiving visual feedback according to said displayed image; and 
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(d) receiving tactile feedback according to said location of said 
endoscope within said gastro- intestinal tract. 



16. A method for displaying simulated visual data of a medical 
procedure, the method comprising the steps of: 

(a) recording actual video data from a performance of an actual 
medical procedure on a living human patient; 

(b) abstracting a plurality of individual images from said actual video 
data; 

(c) digitizing said plurality of individual images to form a plurality 
of digitized images; 

(d) selecting at least one of said plurality of digitized images to form 
a selected digitized image; 

(e) storing said selected digitized image as texture mapping data in a 
texture mapping database; 

(f) providing a three-dimensional mathematical model of an actual 
human colon, said model being divided into a plurality of 
segments; 

(g) selecting one of said plurality of segments from said model for 
display; 
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(h) overlaying said texture mapping data from said texture mapping 
database onto said segment of said model to form at least one 
resultant image; and 

(i) displaying said resultant image. 

17. The method of claim 16, wherein said texture mapping data 
further include animation. 

18. The method of claim 17, wherein said animation includes random 
movement of said simulated instrument and random movement of said 
simulated organ. 
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